Depth maps and images together may constitute the primary input of many applications, such as video surveillance, video games (e.g., the Microsoft Kinect), hand gesture interpretation and other applications that take input unobtrusively from an un-instrumented user. Other related applications that take depth maps and images as input may include those that analyze the 3D environment around a sensor, for instance for autonomous control of a robot or vehicle or a safety monitoring system.
In some cases, the design of such applications may be easier if the depth map and image are registered or aligned, in the sense that the depth map is, or made to appear to be, produced by a depth sensor that is placed at the same physical location as the imaging sensor that produced the image. When this is the case, the pixels of the depth map may be put into correspondence with the pixels of the image, and vice-versa.
In practice, however, the depth map and the image are often produced by different sensors and consequently may be imaged from distinct physical locations. Fortunately, it may be possible to warp a depth map or image in such a way that it appears nearly as seen from a different center of projection.
In the drawings, the leftmost digit(s) of a reference number identifies the drawing in which the reference number first appears.